Searching for sterile neutrinos in ice 

Soebur Razzaque^ff] and A. Yu. Smirnov^j^ 

(1) College of Science, George Mason University, Fairfax, Virginia 22030, USA 
(2) The Ahdus Salam International Centre for Theoretical Physics, I-S^lOO Trieste, Italy 



Abstract 



Oscillation interpretation of the results from the LSND, MiniBooNE and some 
other experiments requires existence of sterile neutrino with mass ~ 1 eV and mixing 
with the active neutrinos |f//iOp ~ (0.02 — 0.04). It has been realized some time 
ago that existence of such a neutrino affects significantly the fluxes of atmospheric 
neutrinos in the TeV range which can be tested by the IceCube Neutrino Observatory. 
In view of the first IceCube data release we have revisited the oscillations of high 
energy atmospheric neutrinos in the presence of one sterile neutrino. Properties of 
the oscillation probabilities are studied in details for various mixing schemes both 
analytically and numerically. The energy spectra and angular distributions of the 
f^— events have been computed for the simplest z^^— mass, and Us — '^fi mixing schemes 
and confronted with the IceCube data. An illustrative statistical analysis of the present 
data shows that in the i/^— mass mixing case the sterile neutrinos with parameters 
required by LSND/MiniBooNE can be excluded at about 3a level. The Ug — Vfi mixing 
scheme, however, can not be ruled out with currently available IceCube data. 
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1 Introduction 



There are several experimental results which could be interpreted as due to oscillations 
related to existence of sterile neutrinos with mass m ~ 1 eV and rather large mixing with 
or/and z/g. This includes the LSND result [T], the MiniBooNE excess of events in neutrino 
and antineutrino channels [2], the reactor antineutrino anomaly [3J and the results of the 
solar calibration experiments |1] (see |5] for recent interpretation). Global analysis of the 
short-baseline oscillation experiments shows certain consistency of different evidences in 
the two sterile neutrinos context [H]. Furthermore, the analysis of CMB data indicates an 
existence of additional radiation in the Universe [7] with sterile neutrino being one of the 
plausible candidates. The effective number of neutrino species, A^^efr ~ 4-5, looks preferable. 
The bound on A^efr from the Big Bang Nucleosynthesis (BBN) has been relaxed recently 
allowing for 1-2 additional neutrinos with the best fit value above 3 species [8] . 

At the same time, the recent global cosmological analysis which includes the CMB data, 
large scale structure and BBN results, shows that existence of new neutrino species does not 
relax significantly the bound on mass of the sterile neutrino [9]. For AA'^gfr = 1 one obtains 
rris < (0.5 — 0.6) eV or Am^ < (0.25 — 0.36) eV^ which is smaller than the LSND-required 
value. 

It has been observed some time ago that mixing of sterile neutrinos with m ~ 1 eV, and 
therefore Am^ ~ 1 eV^, strongly affects the atmospheric neutrino fluxes in the energy range 
500 GeV-few TeV. In this energy range the MSW resonance in matter of the Earth is realized 
in the — Vg or — Us channel \T0\. The resonance enhancement of oscillations leads to 
appearance of a dip in the energy spectrum and to distortion of the angular dependence of 
tracking (i/^— induced) events. These effects can be studied in the IceCube detector [TU] . 
Later in [TT] an extended study of the oscillation probabilities has been performed in the 
presence of one or two sterile neutrinos. As an experimental test it has been proposed to 
measure the ratio of the tracking and cascade (induced by z/g) events. 

Recently AMANDA [12j and IceCube |13j have published the first high statistics data 
on the atmospheric neutrinos in the TeV range. (See also results from SuperKamiokande 
|14j). In this connection we present both analytical and numerical study of properties of 
the relevant oscillation probabilities for different mixing schemes. We compute the energy 
spectra and angular distributions of events in IceCube. Results of these computations are 
confronted with the IceCube data and bounds on the parameters of sterile neutrinos have 
been obtained. We show that observational results substantially depend in the Ug— mixing 
scheme. 

The paper is organized as follows. In Sec. 2 we describe the simplest mixing scheme 
for sterile neutrino (the i/^— mass mixing) for which dynamics of evolution is reduced to 
the 2i/— evolution. We obtain the analytical expressions for the oscillation probabilities 
and present results of numerical computations of the probabilities. In Sec. 3 we study 
modifications of the atmospheric neutrino fluxes due to mixing with sterile neutrinos. We 
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compute the number of events for IceCube and confront them with experimental data. In 
Sec. 4 we perform an illustrative statistical analysis of the data and obtain bounds on the 
mixing of sterile neutrinos depending on the sterile neutrino mass. In Sec. 5 the oscillation 
effects are considered in the z/^ — i/^ mixing scheme. We compute the probabilities and 
zenith angle distributions of the z/^ events, and perform the x^~analysis. In Sec. 6 we study 
dependence of the oscillation effects on the mixing scheme in the leading order approximation 
(valid at high energies). Conclusions are given in Sec. 7. In the Appendix we present explicit 
expressions for the probabilities in the constant density case. 



2 Oscillation probabilities in the z/^— mass mixing scheme 

We will consider mixing of four flavor^ of neutrinos (i^sj ^e-: ^y^^ i^r) which mix in four mass 
eigenstates z/j, i = 0,1,2,3. We assume the neutrino mass hierarchy: mo S> 1713,1712, mi, 
since the opposite situation: m^ ^ m2 ~ mi ^ mg, with three active neutrinos in the eV 
range is strongly disfavored by the cosmological data. The mass squared differences equal 

Aml^ = {ml - mj) ~ (0.5 - 3) eV^ Amj^ = {ml - mj) ^ 2.5 ■ 10"^ eV^ 

as is required by the LSND/MiniBooNE and fixed by the atmospheric neutrino results. 

As we will show, for high energies {E > 100 GeV) the electron neutrino mixing can be 
neglected in the first approximation in consideration of the z/^— , z/^— oscillations. Therefore 
the system is reduced to mixing of the three flavor states uj = {ug, Vr, v^) in three mass 
eigenstates v'^ass = ('^o, '^s, ^^2) as z// = UfUmass, where Uf is the mixing matrix. 

In this section we will consider the simplest mixing scheme when Ug mixes in the states 
uq and z/3 with masses mo and m^ only. In this case 

Here U23 is the usual 2-3 rotation on the angle 6*23 ~ 45° and Ua is the rotation of the mass 
states z/q and on the angle a. Explicitly, 



Uf = f/23f/a 



COS a sm a 



/ 

— sin a cos 6^23 cos a cos 6^23 sin 6^23 | . (2) 
y sin a sin 6'23 — cos a sin 6'23 




The sterile neutrino mixing is characterized by a single new mixing parameter. In what 
follows we will refer to ^ as to the z/^— mass mixing scheme in contrast to the z/^— flavor- 
mixing scheme which will be discussed in Sec. 5. The simplest mixing scheme allows us to 
reduce dynamics of the 3z/— evolution to 2z/— evolution exactly. Other schemes allow to do 
this only approximately. 
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According to ^ Us mixes with the state 

u'^ = cos 6*23 Z^r - sill 6*23 I^/x, (3) 

and there is no mixing of with the orthogonal combination: 

Z/^ = cos 0231^ f, + sin 923l^r- (4) 

Thus, 

= COS a Us — sin a u'^, = cos a + sin a Us, 1^2 = t'^- 

In the first approximation at high energies the dominant effect is due to oscillations driven 
by the largest mass splitting, Aml^. Therefore the transitions are described by the flavor 
mixing in the uq state. The corresponding elements of mixing matrix equal 

Uso = cos a, Uf^Q = sin a sin 6*23, Uqt = — sina cos^^23- (5) 

The mass squared difference Am^g gives sub-leading effects at high energies. But it produces 
the leading effects at low energies {E < 0.5 TeV). 

Consider evolution of this system in the propagation basis defined as 

z>^^(z/„z/:,z/;). (6) 

It is related to the mass basis as z> = U admass ■, and therefore the evolution equation for z/ 
reads 

^^=Hi) = {U^HfU^ + V)v. (7) 

Here ifg*"^ = diag(mg, mg, m2)(2i?)~^, and V = diag(— V^, 0, 0) is the matrix of the poten- 
tials. In V we have subtracted the matrix V^I proportional to the unit matrix I. In this 
way we factor out the 2-3 mixing from the evolution of neutrino system. (For earlier work 
on evolution of 3 and more neutrino states in matter, selection of the propagation basis 
see [15]). For neutrinos in the electrically neutral medium: 



Vf, = Vr = --^GpriNil - Ye) = --^Gprin, 



where n^r = p/frtN is the total number density of nucleons, n„ is the number density of 
neutrons and Y^. is the number of electrons per nucleon in the medium. In the electrically 
and isotopically neutral medium Therefore = 0.5K, where Ve is the 

difference of potentials for the Ue — z^^ system. For antineutrinos: V^j, = —V^. Explicitly the 
Hamiltonian is given by 



H 



f ^cos^a-V, -^sin2a \ 
_^sin2a -^sin^a 



4E '^"^^^ 2E 2 

- ^""32 / 

"J ^ 2E ' 



(8) 
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Here again we have subtracted the matrix proportional to the unit matrix {m'^/2E)I. 
The MSW-resonance condition reads 



— ^cos2a = K 



^l1 



and since < the resonance is reahzed in the antineutrino channeL The resonance 
energy ~ (2 — 5) TeV (Ampg/leV^), (see the level crossing scheme in [IZ|)- The state v'^ 
decouples and is not affected by matter. It evolves independently as 



«032 



■>32 



2E ■ 



(9) 



As follows from the form of the Hamiltonian ([8]) the evolution matrix (matrix of ampli- 
tudes) in the propagation basis can be written as 



S 



(no z/'— transitions). From unitarity of 5* we have: 




(10) 



1. 



(11) 



According to ^ and Q the states of the propagation basis u are related to the flavor 
states Uf as 

Therefore the 5* matrix in the flavor basis Vf is 

S = U2d,SU2^. 



(12) 



Using (|12|) and (10) we obtain 



S 



( A COS 6^23^ sr' 

COS 6'23^r's COS^ ^23^t't' + slu^ 6'23^Af /i' 

\^ - sin 6'23^t's - sin 6^23 cos 6'23(^r'r' - ^/.V 



- sin Q^-iKr' 
- sin 6^23 COS 6^23 (-4^/^/ - A^/^/ 

Sin^ 6'23^r'r' + COS^ 6'23^^>' 



(13) 

Moduli squared of the elements of this matrix give the corresponding oscillation probabilities. 
In particular, the — z/^ survival probability, P^^, equals 

2 



P, 



MM 



sin 6'23^r'T' + COS Q2?,Au 



(14) 



and the other oscillation probabilities with participation of z/^ are 



P 



Sin^ 6^23 1 ^r's 



sin^^^23(l - I^t't'I 



p 



sin^ 6*23 cos^ 023] At' t' — A, 



(15) 
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These probabilities satisfy the unitarity condition: P^^ + P^,- + P^^ = 1. Notice that when 
At-'t-i = —A^'^', the transition probabihty P^r = PJIi"'^ = sin^2^23- In this case P^^ = 
cos^ 2^23- Then for maximal 2-3 mixing we have P^^ = and P^r = 1, and correspondingly, 
P>.s = 0. 

Consider properties of the survival probabilities in the neutrino and antineutrino chan- 
nels. Using ^ and ( 14 ) we obtain 



p 



sin" ^23^r'r' + COS^ ^236*'^^' 

sin^ ^23i'rV' + 2 sin^ ^23 cos^ ^23^6 (e-'^^'A,,,,) + cos^ ^23, (16) 



where Pr'r' = l^rvl^ is the i>'^ — u'^ survival probability. For antineutrinos we have A^/^/ — )■ 
A^,^, = Ar'r'iVf, -V^). For E > 1 TeV the 2-3 phase is small, (^32 < 3° - 4°, so that in 
the lowest-order approximation 

P^^ ^ Sin^ 923Pr'r' + 2 siu^ ^23 COS^ ^23^6 {A^'r') + COS^ ^23- (17) 

However, the phase 032 can not be neglected at low energies, E < 0.5 TeV. Explicit analytic 
expression for the amplitude Ar'r' is given in the Appendix. In the absence of mixing with 



sterile neutrino one has a = 0, = 1, and consequently (16) is reduced to usual vacuum 
oscillation probability due to the 2-3 mixing and 2-3 mass splitting. 

In Fig. [l] we show the probability P^^ as a function of neutrino energy for different 
values of the zenith angle {6z) and the oscillation parameters. (In our computations we 
use the PREM model for the Earth density profile [IH]-) The typical energy-dependent 
feature of P^^ is the resonance dip in the range determined by the resonance energies in 
the core and in the mantle. For | cos6'2| < 0.82 there is a single dip at ii^ ~ Er ~ 4 TeV 
which corresponds to the MSW resonance in the mantle of the Earth. For | cos 6*2! > 0.82 
(core crossing trajectories) the dependence of the probability on E is more complicated. 
The dip between the resonance energies in the core and mantle is due to the parametric 
enhancement of oscillations, i.e. due to an interplay between the oscillation effects in three 
layers with nearly constant density (mantle-core-mantle) [19] . The width of this dip is larger 
than the width of the MSW dip in constant density medium. There is also the parameteric 
enhancement of the oscillations at energies above the resonance energy in the mantle |19j . 

For the u— (non-resonance) channel, the peaks are absent (see Fig. |2]), but another 
feature related to the matter effect is realized: enhanced /i — r transition at low {E < 0.5 
TeV) energies. The survival probability decreases with energy in contrast to the u channel 
where P^^ increases with energy. The reason can be understood from consideration in the 
case of constant density (Appendix). At energies below 0.5 TeV the oscillations induced by 
the 2-3 mixing and mass splitting become important. The dependence of probabilities on 
energy is given by the oscillatory curve with low frequency in the energy scale and the depth 



sin^2^23 ~ 1 (see analytic expression in (51)). This curve is modulated by high frequency 



oscillations driven by Ampg with small depth. At low energies the phase of the low frequency 



5 



oscillations is given (see (52) in the Appendix) by 



where in last expression the first term is due to the matter effect; the plus sign corresponds 



to neutrinos and the minus sign to antineutrinos (according to (48), ~ — V^sin^a). 
In the energy interval (0.1 — 0.5) TeV the two contributions are comparable. Thus, the 
matter effect produces an opposite change of the phase velocity: increasing the velocity 
in the neutrino channel and decreasing it in the antineutrino channel. Consequently the 
oscillations due to the 2-3 mass splitting and 2-3 mixing develop in the u— channel at 
higher energies. Notice that the phase shift is proportional to sin^ a, and at low energies 
am ~ a (see Fig. [2| the upper panel). 

Let us consider more general situation when Ug mixes also in the 1^2 state. We introduce 
an additional rotation in the 1^3-2/2 subspace, so that the propagation basis becomes: 
u = UaU^Vrnass- Explicitly the mixing matrix in the propagation basis becomes 

(Cq, SaC^ S/ySy \ 
— Sa CaC^ CaSy . (18) 
—Sj j 

Here = sin 7, = cos 7, etc.. Now Ug mixes in all three mass eigenstates: 

Us = Cal'o + S„(c^I/3 + S^Z/2). 

The Hamiltonian in the propagation basis equals = UaU^H'^^"-^U^U^ , and it can be 
represented as 



where H is the Hamiltonian without rotation ([8j). The correction is proportional to 
a small quantity s^-^^ which produces even smaller (suppressed by s^) phase than 



considered in the simplest case above. (The matrix in (19) is symmetric and elements 
denoted by dots equal to the corresponding transponent elements.) So, the effects of Us 
mixing in i>2 can be neglected in the first approximation. 

The mixing matrix in the fiavor basis is given by Uf = U23UaU.y. The elements of this 
matrix which describe oscillations with large mass split Arngg (dominant at high energies) 
are the same as in our simplest mixing case ([5]). They do not depend on 7. 

In what follows we present predictions for IceCube in the simplest i/g— mass mixing case. 
Consideration of the i/^— fiavor mixing schemes is given in Sec. 5, where we show that, in 
fact, the probabilities and observables substantially depend on the mixing scheme. 
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3 Fluxes and numbers of events 



The z/^— flux at the detector equals 

where and $g are the original fluxes of z/^ and z/g without oscillations. Similar expression 
holds for the antineutrinos. The effect of Ue — ?■ z/^ oscillations can be neglected (the last 



equality in (20)). The reason is two fold: at high energies $° ^ $g, with ratio r = > 



20 for ~ 1 TeV. Furthermore, the transition probability P^^ <^ 1 and can be mostly 
converted to z/^. 

Let us consider z/g oscillations in some details. At high energies the mixing of z/g and 
z/^ is strongly suppressed: sin^ 2^i2(-E'f^/-E')^, where ~ 0-1 GeV is the resonance energy 
associated to the "solar" mass splitting Am2i- The z/g — z/^ mixing is absent in the limit 
6'i3 = 0, but if non-zero, the 1-3 mixing in matter is also suppressed in the TeV energy 
range as ~ siia^ 29i3{E^^/ EY , where E^^ ^ 6 GeV is the energy of 1-3 resonance. Consider 
the whole Au— scheme with z/g admixture, Ueo, in the state z/q. Since for the z/g potential 
we have ~ — in the isotopically neutral medium, the z/g — level crossing is in the 
neutrino channel. The corresponding resonance energy E^^ ^ E^^. The depth of z/g — 
oscillations driven by Amgi equals 

^eM^4|[/g-nf/-r, (21) 



where f/Jj]" and U"^ are the mixing parameters in matter. In vacuum: D^,^ = sin^ 26'lsnd 



3-10 ^. The mixing and the depth can be enhanced in resonances. In the z/^ — Ug resonance 



the z/^— mixing is enhanced, ~ 1/2; whereas the z/g— mixing is suppressed: \U'^q 

|f/eoP/4. As a result, Dem ~ |f^eoP/2 ~ 0.02. In the z/g — z/^ resonance, inversely, the 
z/^— mixing is suppressed ~ |t/^oP/4, and z/g— mixing is enhanced = 1/2. So 

that the depth of oscillations equals D^^ ^ |f/^oP/2 < 0.02. Therefore Pg^ < 0.02, and the 
contribution of the original z/g flux to flux at a detector, Pe^r, is smaller than 10 

The rate of z/^ events in a detector such as IceCube is given by 
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A^ 



JdEjdn [$^(E, e,)A,s{E, e,) + <l^(p, e.)Aeff(^, ^.)] , (22) 



with the appropriate integrations over the neutrino energy and solid angle. Additional 
contribution to the muon events comes from the z/^ — ?■ Vr oscillations, producing a flux 
$T- = $°-P/iT at the detector. The tau lepton from v-r interaction has ~ 18% probability to 
decay into muon, which is then recorded event. The z/^ energy, however, needs to 

be ~ 2.5 times higher than the energy to produce muon tracks of the same energy in 
the detector. Notice that in the z/^— mass mixing scheme z/^'s appear in the z/^ oscillation 
dip, but this will lead to additional events at low energies. In other mixing schemes z/^'s are 
transformed mainly into z/^'s, and production of Ur is suppressed. 
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In (22) Acs and A^s are the effective areas of the detector for u and u. They are given 
by the effective volume Ves from which the events (muons) are collected with an efficiency 
of detection edet as 

Here tt-at is the number density of nucleons in the surrounding medium and a^N is the 
neutrino-nucleon charge-current cross section. In turn, Ves is determined by the geometry 
of the detector and the muon range R^: Ves oc R^. The range can be estimated as 

1 a + tEM-iy)) 



^ 6 a + hE 



where a = 0.24 GeV m~^, b = 3.3 ■ 10~^ m~^, (y) is the mean inelasticity and E^^rnin is 
the minimum muon energy for detection. At low energies R^ oc E^ and at ~ 1 TeV the 
linear increase of R^ changes to the logarithmic one (see [13] for details). Since, usually the 
data are presented using energy bins of equal size in the log— scale, the relevant quantity 
which determines the number of events in a given energy bin is Ne = AceE^^^, (where the 
E originates from the Jacobian). The differential neutrino flux decreases as oc 
and therefore at low energies Ne increases as Ne oc E^'^. It reaches maximum at ii^ ~ 0.7 
TeV and then decreases since Kg has only logarithmic increase. The median energy interval 
E = (0.15 — 2.3) TeV is determined by a condition Ne > 0.5N^'^^. This interval includes 
the region of dips in the oscillation probability and therefore IceCube is well optimized to 
search for sterile neutrinos with Am^ = (0.5 — 2) eV^. The described dependence of Ne on 
energy allows one to understand various features of the predicted effects. 

The effective area is also given by 

^eff = A(^etSEa.Tth{E^,9z)Pmt{E^), 

where A^et is the geometrical area of the detector, S'sarth is the survival probability of 
neutrino passing through the Earth at a given trajectory and Pint is the charged current 
neutrino-nucleon interaction probability in the vicinity of the detector. The survival prob- 
ability is given by 



SEarth{Ei,,9z) = exp 



-NACTtotiE,) [%{e,,l)dl 
Jo 



where L = 2_REarth cos 6^^ is the length of the trajectory, p{9z,l) is the matter density at a 
distance / along the trajectory and cxtot is the total neutrino cross-section. For E < 10 TeV, 
'S'Earth ~ 1- The interaction probability is given by 

Pmt{Eu) = NA(TyN{.E^){R{Ey,E^^^i^)), 

where {R{E^, -E^,min)) is the average muon range in the medium and A^^ is the Avogadro's 
number. 
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In Figs. [3] and |4] we show the sum of the and z/^ energy spectra integrated over the 
sohd angle for the z/g— mass mixing scheme. An estimation of the size of the oscillation 
effects is rather easy: maximal, ~ 100%, effect is for u in the resonance range; summation 
with u (whose flux is about 1.4 times larger) gives 40% effect; averaging over the zenith angle 
from 180° to 90° produces another factor ~ 1/2, and therefore one arrives at the maximal 
~ 20% suppression in the dip. Relative effect increases with narrowing the integration 
region around vertical direction (see Fig. [s]). Now the maximal effect can reach 40% and 
further enhancement would require experimental separation of neutrino and antineutrino 
signals. With increase of Am-Qg the dip shifts to high energies as E oc Arrigg. Increase of the 
size of the dip with sin^ a is more complicated. Suppression effect extends to low energies 
due to oscillations in the u— channel driven by the 2-3 mixing. 

We also compare the predicted neutrino energy spectra in Figs. |3] and |4] with the "un- 
folded" energy spectra reconstructed by IceCube [13]. Presently, this comparison can be 
used for illustration only since reconstruction of the unfolded spectra implies significant 
smearing and in general is not sensitive to the spectral distortion in small energy intervals. 
Notice, however, that the size of the dip in the energy scale is larger than the size of the bin 
of the reconstructed spectrum. To have better sensitivity to the distortion one can further 
decrease the size of the bin. 



According to the Fig. 22 of [13] the statistical error in the relevant energy range is about 
3% which is substantially smaller than the size of the dip. Continued operation of IceCube 
in future will reduce this error further. Large errors are due to systematics: mostly due to 
uncertainties in the total normalization and tilt of the spectrum. To a large extent they can 
be eliminated when searching for the dip. Indeed, the systematics has smooth dependence 
on energy, the systematic errors in different bins are strongly correlate. One can parametrize 
these uncertainties by a few parameters and determine them by fitting data. 

The problem of smearing does not exist in the case of the zenith angle distribution, 
since muons nearly follow neutrinos, and the zenith angle resolution is 0.5 — 1°. We compute 



the number of events Nj in a given zenith angle bin AjcosOz using (22) and performing 
integration from the threshold Efh'- 

Nj = 2nf dcose, f dE <!>1{E, 6,) A^siE, 6,) P^^{E, 6,) + antmeutrinos. (23) 
We then define the suppression factors in the individual bins as 

= ^> (24) 



where are the numbers of events without oscillations which correspond to P, 



in (23). In Figs. |6j and [7| we show the zenith angle dependence of the suppression factor 
for different values of the mixing parameter sin^ a and sin^ 6*23 = 1/2 (this corresponds 
to |t//xoP = O.Ssin^a) and two different thresholds Eth = 100 GeV (Fig. |6| and Eth = 1 
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TeV (Fig. [7]). Oscillations lead to distortion of the zenith angle distribution. For nearly 
horizontal direction the effect is mainly due to vacuum oscillations which have enough 
baseline to develope if E < 0.5 TeV. In this case the averaged oscillation effect is given 
by 1 — 2|[/^oP(l ~ If^/iOp) ~ 1 — sin^a in agreement with the results of Figs, [g] and [Tj 
The matter effect increases with | cos6z\. According to these figures substantial differences 
between the energy-integrated distribution with and without sterile mixing are expected 
in the bins near the vertical direction. For sin^ a = 0.04 the effect is about 20% and the 
statistical errors, 3%, are much smaller. For other mixing schemes the distortion can be 
different. In particular, in the i/,,— flavor mixing scheme maximal suppression is in the bins 
cos^^ = (-0.9,-0.8) (see Sec. 5). 

For vertical directions the evaluation of the suppression (integrated over the energy) 
can be done using the survival probabilities of Figs, [l] and [2] If e.g. sin^ a = 0.08, the 
probabilities averaged over the median energy interval in the neutrino and antineutrino 
channels are P^^ = 0.6 and P^^ = 0.8 respectively. Then averaging the contributions of the 
neutrinos and antineutrinos we obtain S ~ (P) = 0.70 — 0.75, in agreement with results 
in Figs. [6] and [7} With increase of threshold, the effect of vacuum oscillations in nearly 
horizontal directions becomes smaller. The effect in the u channel increases, whereas in the 
u channel it decreases, thus compensating the overall change. 

In Fig. [8] we confront the experimental results with the predicted zenith angle distribu- 
tions computed as 

N, = Nf^S,, 

where Nj'^'-' is taken according to the IceCube simulation (see Fig. 19 from [I3]). We have 
implemented an overall normalization and tilt of the distribution, as we discuss below in 



4 Bounds on parameters of sterile neutrinos 

To get an idea of the sensitivity of the currently available IceCube data to the sterile neutrino 
mixing we have performed a fif of fh^ IceCube zenith angle distribution. For a given 
"model" of mixing characterized by {ArriQ^, sin^ a) we compute the expected number of 
muon events Nj^"'^ in the zenith angle bin j. For this we use the IceCube simulation, Nj^'~' 

N]^°'^{C,T;Aml^,sma) = C[l + r{cos9j + 0.5)]Nj^^ Sj{Aml^,sm'^ a) , (25) 

where C is an overall normalization parameter and r is a zenith angle tilt parameter. The 
model without mixing is recovered when a = 0. We compare the expected numbers N^°'^ 
with data N^^^ and the is defined as 

X\C, T- Ami,, sin «) = E — 2 ^ • (26) 

/ ^dat 1 
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The variance aj is calculated by adding in quadrature the statistical and systematic un- 
certainties as given by IceCube \13]. For our analysis we use the IceCube data in the range 
of zenith angles —1 < cos^^ < —0.1 {i.e., bins j = 1-18), leaving out the last two near hor- 
izontal bins where the detector response is not well-understood and contamination of the 
atmospheric muons is possible. For fixed values {Aml^,sma) we minimize the varying 
the (C, r) parameters. The difference 

^X^ = xLiniC, r; Am^3, sin^ a) - xLniC, r; Aml^, 0) 

quantifies the rejection significance of the Ug mixing model with respect to the model without 
Us mixing. 

In Table 1 we show results of our statistical analysis which is reduced to determination 
of the minimal values of C and r for given Amgg and sin^ a. We show Xmm the best 
fit values of C and r for the case of statistical errors for the individual bins only (see Fig. |8]). 
Also shown is the fit for the "null" hypothesis. Notice that the mixing sin^a ~ 0.01 fits 
the data better than the model without Us mixing ("null" model) Ax"^ < 0. Also notice 
that for sin^ a ^ 0.04, C and r are below 3% and then they quickly increase with a reaching 
12 - 13% for sin^ a = 0.08. 

Fig. [9] (left panel) shows the bounds on the sterile neutrino mixing as function of Amgg 
from the analysis which takes into account statistical uncertainty in each bin as well as the 
systematic uncertainties due to overal normalization and tilt of the zenith angle distribution. 
These are the main uncertainties. To illustrate possible effect of other systematics we have 
taken the extreme case: 5% uncorrelated errors for individual bins (see Fig. |9| right panel). 
(Although it is expected that other possible uncertainties are smooth functions of the zenith 
angle and therefore correlate in different bins.) In reality the effect of additional errors should 
be smaller than that. The parameter space to the right hand side from the lines in Fig. |9]is 
excluded at the indicated confidence level. 

The bounds weakly depend on the Arngg, as can be seen from the behavior of the 
suppression factors (Figs. [6]and[7]). The bounds are slightly weaker for smaller Arngg since 
in this case the resonance dip shifts from the energy range where IceCube has the highest 
sensitivity. 

We find that with statistical uncertainties only (Fig. [o] left panel) the upper bound is 
sin^ a < 0.05 or lU^ol"^ < 0.025 at 3a level and Aml^ = 1 eV^. At 2a level the bounds are 
sin^a < 0.04 and |f/^oP < 0.02. At the same time interpretation of the LSND/MiniBooNE 
results in terms of oscillations in the presence of sterile neutrinos requires |?//iOp ~ 0.03 for 
Am^3 = 1 eV2, and \Uf,o\^ > 0.06 for Amgg = 0.5 eV^. 

With 5% uncorrelated systematic errors (Fig. [9] right panel) the limits become substan- 
tially weaker: sin^a = 0.06, is excluded at 90% C.L. only. 
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Table 1: Results of the x^~cinalysis of the IceCube zenith angle distribution. Shown are 
Xmin ^^11 best fit values of the normalization parameter C and tilt r for given 

values of Arngg and sin^ a in the mass-mixing scheme. 
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5 Oscillation effects in the — mixing scheme 

Let us consider the Vs — mixing only, i.e., the simplest scheme of z/^— flavor mixing. The 
corresponding mixing matrix in the flavor basis (z/^, z/,-, z/^) equals 

^ C24 — 524-523 ■524C23 \ 
Uf = U2dJ23 = C23 S23 , (27) 

V —S2A ~C24S23 C24C23 / 

where S24 = sin ^24, eic. Formally it differs from the mixing in ([!]) by permutation of U23 
with the z/g— mixing matrix. 

Now the mixing matrix elements, which determine the oscillations with splitting Arngs, 
equal Ugo = C24, Uro = 0, f/^o = — S24. They are reduced to the elements of our simplest case 
([5]), if formally we take C23 = and S23 = — 1 and a = 92a- Therefore in the leading order 
approximation for high energies the probabilities can be obtained from the probabilities in 
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z/g— mass mixing case by taking S23 = —1. In particular, according to (14) 



724 J 



(28) 



For AT-ir' = — 1 we obtain = 1, whereas in the i/^— mass mixing scheme this value gives 
the minimum of the dip P(""^**) = Q. 

It is possible to find relation between the sizes of dips for different mixing schemes. 
For maximal 2-3 mixing we have from ( 14 ) the survival probability in the Vs —mass mixing 
scheme (|2]): 



ry{mass) 



\Ar'r' + 1| 



(29) 



In the resonance, the amplitude At-'t-i is approximately real. This can be seen using explicit 
results for the constant density case. Indeed, according to (48) in resonance Him = — -f^2m, 
and therefore (49) gives Ar'r' ~ cos{Himx). Then from (29) and (28) we obtain relation 



between the probabilities: 



pif) 



2\/P 



(mass) 



(30) 



Our numerical results in Fig. 15 confirm this relation. 



Let us consider corrections to the leading order result due to oscillations driven by the 2-3 
mixing and splitting. They are sub-dominant at high energies, but become dominant at low 
energies. In the i/^— fiavor mixing case it is convenient to consider oscillations immediately 
in the fiavor basis, i. e. take the fiavor basis as the propagation one. Using the mixing matrix 
(27) we find the Hamiltonian of evolution H = U fH'^^"'^Uj + V which can be represented in 
the following form 



H 
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• (31) 



At high energies the evolution is described by the first term of the Hamiltonian (which does 
not depend on the 2-3 mixing), decouples and the corresponding S matrix in the fiavor 
basis can be written as 



S 



^ Ass 





As^ 





1 










A^^i 


= ^4^At 


12 
1 1 


is in 



(32) 



So that the survival probability, P^^ = is in accordance with (28). Indeed, the 

first term of the Hamiltonian (31) coincides with the Hamiltonian ([S]) up to permutation 
of the 2-3 lines, 2-3 columns and substitution a — )■ 6'24, and therefore = At'ti in this 
approximation. With the sub- leading term of the Hamiltonian taken into account, evolution 
is not reduced to the 2 //—evolution. 
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Effect of the 2-3 mixing at low energies {E < 0.5 TeV) can be estimated in the following 
way. In the basis z/^ defined in such a way that uj = U24i'a the Hamiltonian is given by 



'24 ' 



^24 ; 



or explicitly 
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23 2E 
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For energies much below the sterile resonance, Vn <^ 



Ami 



one can perform a block diago- 



M 2E 

nalization thus decoupling the heaviest state, or simply neglect the 1-3 terms S24C24V^ in the 



Hamiltonian (33 ). The latter is equivalent to an approximation of negligible matter effect on 
the angle 624- So, the evolution is reduced to 2u— problem. Similarly to our consideration 
in Sec. 2 we find (returning to the flavor basis) that the 



P, 



MM 



-24 



_|_ e2 A (a) 



MM 



Ufj, survival probability equals 

(34) 



where A^J^^ = exp[— 2x(^^ "^24^^)], and the amplitude A^^^l should be obtained by solving 
the evolution equation with the Hamiltonian 





sin 2^23^ 



Sm Zt)23^^ 
Am? 
'23^ 



COS 26^23^^ + si4^M 



(35) 



Here we have subtracted from the 2x2 submatrix of (33) the matrix proportional to the 

z/^ probability averaged over fast oscillations driven by Ampg equals 



unit matrix. The z/„ 



P, 



-24 



-24 



I^M^- 



(36) 



The matter effect on the amplitude A^^^ becomes substantial when 



~ «24I'^'mI' 



i.e., E ~ 150 (O.O4/SI4) GeV. Matter suppresses the depth of z/^ — ly^ oscillations and 
increases the phase velocity as compared to the vacuum oscillation case. For the maximal 
2-3 mixing the effect is the same in the neutrino and antineutrino channels: 



32 = xJ{Aml2/2Ey + {sl,V, 



For non-maximal 2-3 mixing the resonance is realized at 

E -- 



2^ cos 2^, 

^■'24 *^M 



23) 
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and the picture becomes v — v asymmetric depending on cos 2^23- We find that for ^23 = 
7r/4, S24 = 0.04 and cos6'2 = —1.0 the averaged (over fast oscillations) corrections to the 
probabilities in both channels equal AP^^ 0.15 oX E = 100 GeV and ^Pfi^j. ~ 0.02 at 
E = 300 GeV. 



Results of numerical computations of the probabilities shown in Figs. 10 and 11 confirm 
this analytic considerations. Qualitatively the probabilities as functions of the neutrino 
energy look rather similar to those in the i/g— mass mixing scheme. As we discussed, certain 
difference appears at low energies. We find also that at sin^ 6^24 = 0.08 the dip for cos 9z = —1 
is suppressed and maximal suppression is achieved at cos6z = —0.90, in contrast to the 
mass-mixing case. Also here the size of the dip decreases slower with increase of cos 
This result holds for bigger mixing angles: If sin^ 624 = 0.08, in the vertical bin we have 
PI/^{cos9z = —1) ~ 1, and maximal suppression in the dip, P^j} = 0, is achieved at 
cos ^2 = —0.80. Here in the dip region is transformed mainly to Ug. So, the appearance 
of Ur is the signature of the i/<j— mass mixing scheme. 



In Figs. 12 and 13 we present the zenith angle dependence of the suppression factor 
for the muon events integrated over the energy from Eth = 0.1 TeV and Eth = 1 TeV, 
correspondingly. We compute these dependences in the same way as we did for the z/^- mass 
mixing scheme. Notice that for sin^^24 < 0.04 the distributions are flatter than in Figs, [g] 
and [7j The suppression is somewhat stronger in vertical and nearly vertical bins and it 
is weaker in the horizontal direction. In contrast to the previous scheme the distribution 
changes with the threshold more strongly. For sin^ 624, = 0.08, which is essentially excluded 
by MINOS result, a wide dip appears in the range cos = (—0.8, —0.4) (see discussion in 
Sec. 6). 

In Fig. [14] we show the zenith angle distributions of the /i— events. The distributions 
are very similar (with some small deviations in the vertical and horizontal bins) to those in 
the null hypothesis case. 

In Table 2 we present results of the analysis of the zenith angle distribution for the 
Us — t^ti mixing scheme. In contrast to the i/^- mass mixing case, now better fit than in null 
hypothesis case can be achieved for values of sin^ 6'24 = 0.02 — 0.04 and Aml2 = (0.5 — 2) 
eV^ which can provide an explanation of the LSND/MiniBooNE results. So, Ug with these 
parameters can not be excluded by the present IceCube data. 



6 Oscillation effects for generic z/^— mixing in the lead- 
ing approximation 

Let us consider the generic i/g- flavor mixing. The mixing matrix can be written asUj = 
U34U2AU23, where t/34 is the matrix of rotation in the Ug — v-r plane on the angle ^34. The 
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Table 2: Results of the "analysis of the IceCube zenith angle distribution. Shown are 
Xmin as ^^11 as the best fit values of the normalization parameter C and tilt r for given 
values of Arngg and sin^ 624 in the Ug — z/^-mixing scheme. 
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matrix elements which describe the flavor content of z/q equal 

^sO = C34C24, f/rO = — -534024, = " ■S24- 



(37) 



The v^J^ oscillations in vacuum (LSND/MiniBooNE) are determined by the parameter [/^q 
- the admixture of the muon neutrino in the heaviest state. In matter at high energies the 
phase 032 is small and can be neglected, then the relevant parameters are t/^o, ^to, 
The Hamiltonian can be written as 



IE 



where = (f/^o, ?7t0) f^^o); and in the first term we have the matrix formed by the product 
of the column Vq and the line . 

Comparing (37) with the elements in ([s]) we find that the dominant oscillation results 
at high energies in the flavor case can be obtained from the results in the z/^— mass mixing 
scheme identifying 

C34C24 = Cq,, S24 = — ■5aS23, 534024 = Sq,C23. 



That is, in general according to ( 14 ) the probability equals 



MM 



sin^ /3 AriTiio) + cos^ /3 



(3^ 



16 



where Cq = C34C24 or 
and 

Explicitly, 



2_|_2_2 2^2,2 
"^24 + "^34 "^34"524 ^ ^34 ' "^245 



p ~ 



■'24 



^24 



"524 + ■^34 ■534-524 -524 + -534 



(1 £24034)2 



S24.-At't' ~I~ "534^-24 



(39) 
(40) 

(41) 



In Fig. 15 (top and bottom panels) we show the survival probabilities as functions of 



energy for fixed value S24 = 0.04 (as is required by LSND/MiniBooNE) and different values 
of sin^ (3. Notice that for the core crossing trajectories with change of mixing scheme the size 
and form of the oscillation dip changes significantly. The z/^— mass mixing case corresponds 
to sin^/3 = 0.5 or S24 ~ S34, whereas the — i/^ mixing case is realized when sin /3 — 1, 
that is S34 = 0. Recall that at low energies the sub-leading effects due to become 
important. 



In Fig. [16] we show the zenith angle dependence of the suppression factor integrated 

0.04 and different values of sin^ (3. Starting 



over the energy (see definition in (24)) for s 



'24 



from sin /3 = 1 (s§4 = 0) and reducing it down to 0.08 one obtains first fiat distribution, 
then the distribution with a dip at or near the vertical directions and then again rather fiat 
distribution. In all the cases the suppression weakens in the bins close to horizon. 



The dip is at | cos 6*2! > 0.8 in Fig. 16, Indeed, in the z^, 



sion P^fj_ = corresponds to A^-v' 



mixing case maximal suppres- 
0. For the mantle-crossing trajectories (| cos^^l < 0.8) 



this can be achieved if the MSW resonance condition and the oscillation phase condition 
003 = TT are satisfied simultaneously (see also discussion in [H]). The conditions can be 



rewritten as 



^ cos 26124 = V^, 2x = — — 



sin 26*24 

where is the oscillation length in vacuum, x = 2Re \ cos 6z\ is the length of neutrino 
trajectory {Re is the radius of the Earth), and the expression in the RHS of the second 
equality gives the oscillation length in resonance. From these conditions, excluding we 
find 

COS^J = .rr^^ — . (42) 



2REVf, tan 2^24 

Thus, a shift of the dip to small | cos 6*2! would require large z/^ 
latter is restricted by MINOS experiment [2TJ: sin^ 2^24 < 0.14 



z/^ mixing angle ^24- The 
{) C.L.), and for the 



allowed values of ^24 the condition (42) can not be satisfied (see also [22] )• Large mixing 
a in the 2z/ amplitude Ar'r' is possible if S34 is large. However, in this case also sin^ (3 

-cot2/3. 



is substantially below 1. According to (38), P^^ = corresponds to Ar'r' {a) = 
i.e. negative amplitude. In turn, this requires even bigger phase than in the previous case, 
003 > 77, which can not be achieved. 
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Notice that for values of oscillation parameters 



Am^3 ~ (0.5 - 1) eV^ sin^ a ~ 0.04, sin^ /3 ~ 1, (43) 

the zenith angle distribution (suppression factor) for | cos^^| > 0.1 is rather flat in spite of 
profound and wide dips in the oscillation probabilities. A shallow dip in the suppression 
factor can appear in the interval of cos 6*2 (—0.95, —0.8) for AmQ3 ~ 0.5 eV^. For | cos^^l < 
0.1 the suppression becomes weaker which one can still use to disentangle the oscillation 
effect and normalization of spectrum. This flatness of the energy integrated distribution is 
due to (i) specific dependence of the IceCube sensitivity on energy and (ii) correlated change 
of properties of the oscillation dip with change of 9z which is realized for the parameters 



(43) 



The zenith angle distribution with parameters (43) could give even better fit, with a 
decrease in x^^^ value by 3, of the observed distribution than the null oscillation hypoth- 
esis. Furthermore the required values of the overall normalization, 1.057, and tilt, 0.0136, 
are small. The contribution from low energy oscillations driven by 2-3 mixing and mass 
splitting, however, has strong dependence on the zenith angle, and consequently, distorts 
the distribution near vertical directions. Apparently study of the zenith angle distributions 
with different energy threshoulds or in different energy intervals will enhance sensitivity to 
oscillation effects. 

Thus, apart from special case of z/^- flavor mixing in the leading order approximation, 
the allowed mixing schemes predict the dip in the zenith angle distribution in the vertical 
or nearly vertial directions, and therefore are disfavored by the present IceCube data, as in 
the illustrative analysis in Sec. 4. 

Let us compare our results with those in Refs. [10] and [H]. In [10] the flavor mixing 
has been considered with = 0.045 and = 0.45 {i.e., with nearly maximal Ug — Vt 



mixing). According to (|40l) these parameters correspond to sin^ /3 = 0.095, and consequently 



Pun ~ |0.095v4^/^/ + 0.905] . This leads to ~ 10 - 20% effect with weak dependence on the 



zenith angle and energy (see Fig. 4 in \IQ\)- Furthermore, since s\ ~ 0.5 (see (39)), the 
mixing is nearly maximal and therefore the resonance dip is absent (see Fig. 3e in [lO]). In 
|llj the i/g- flavor mixing is considered with = s'^^ = 0.04. This equality means that in 
fact the i/<j— mass mixing is realized with ~ 2s|4 = 0.08. Our results are in agreement 
with those in Fig. 5a of [llj. Our interpretation of the dip at cos 6*2 = —1, however, differs: 
the dip is due to parametric enhancement of oscillations, rather than the MSW oscillation 
dip in the medium with averaged density. 

Thus, the zenith angle and the energy distributions of events substantially depend on 
details of the mixing scheme, and in particular on mixing of Ur in determined by S34. 
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7 Conclusions 



1. We have considered the neutrino oscillations in the Earth in the presence of single sterile 
neutrino with mass m ~ 1 eV [AitLq^ = (0.5 — 3) eV^]. 

2. We present an analytic study of the oscillation probabilities which allows one to 
understand features of dependences of the probabilities on various parameters, and in par- 
ticular, on the mixing scheme. We have identified the simplest mixing scheme in which the 
flavor evolution is reduced to the 2i/ evolution. 

3. The main features of probabilities (in the i^'^— mass mixing scheme) include the 
resonance dips (peaks) in the i>— channel in the energy range (0.5 — 5) TeV: the MSW 
resonance peak for the mantle crossing trajectories and the parametric enhancement peak for 
the core crossing trajectories. In the i/— channel at E < 0.5 TeV there is the matter enhanced 
/i — T transition due to oscillations induced by the 2-3 mixing and mass splitting. The phase 
velocity in the neutrino channel is enhanced due to matter effect, so that oscillations are 
developed already for E = (0.1 — 0.5) TeV. In contrast, in the antineutrino channel, matter 
suppresses the phase velocity and oscillations are not developed. 

4. Oscillation effects on the z/^— and z/^— atmospheric neutrino fluxes and on the sum of 
the fluxes are studied. We have computed the energy spectra of these neutrinos, integrated 
over various zenith angle intervals. Maximal oscillation effect is ~ 40% suppression of the 
flux in a wide (half an order of magnitute) energy interval. The position of the dip is 
determined by the value of A.ml^. The dip has low energy tail due to oscillations driven 
by the 2-3 mixing. In the range (0.5 - 5) TeV the effect is mainly due the resonance dip in 
the //—channel, whereas in the range (0.1 - 0.5) TeV it is mainly due to — Ur oscillations 
with matter modified frequency in the i/— channel. Changes of the energy threshold does 
not modify results substantially. 

5. We have computed the zenith angle distributions of muon events (induced by 
and Pn) in the IceCube detector. Oscillations lead to typical distortion of this distribution 
with about (1 — sin^ a) suppression in the directions close to the horizon, and stronger 
suppression in the directions close to vertical |cos^^| > 0.7. For the i/g— mass mixing 
scheme, the maximal suppression, 20 — 25%, is in the vertical direction. 

6. The relative oscillation effect on the energy spectrum of neutrinos can be enhanced 
by making integration over directions near the vertical one. 

7. We confronted the computed distributions with the IceCube data and performed fit 
of the zenith angle distribution for the i^s— mass mixing. We find that with statistical errors 
and systematic uncertainties in the total normalization and tilt the values, |[/^oP > 0.025 
are excluded at more than 3a level. The central value required by LSND/MiniBooNE is 
|f^AjOp ~ 0.03 is excluded at the Scr level. With additional 5% uncorrelated systematic 
uncertainties the limits become much weaker. 

8. In the case of i/g — J^fj. mixing scheme both properties of the resonance dip and low 
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energy behavior of the probabihties are modified in comparison with those in the z/^— mass 
mixing scheme. We find that maximal suppression is in the bins cos^^ = (—1.0,-0.8). 
The oscillation effects due to 2-3 mixing appears at E < 0.3 TeV both in neutrino and 
antineutrino channels, and the effects are equal for maximal 2-3 mixing. Rather flat zenith 
angle distribution can be obtained in pure z/^ — Vs mixing case with |f/^oP ~ 0.04 and 
AmQ3 ~ 0.5 eV^ as well as for large Ur — Vs mixing: |f/roP ~ 0.5. Fit to the zenith angle 
event distribution substantially improves for this case and gives a better description of data 
than the no Vs~ niixing case. 

9. We have studied the oscillation effects in generic z/^- flavor mixing scheme in the 
leading order approximations valid for high energies E > 0.5 TeV. We showed how results 
for these schemes can be obtained from the results of the Vg—mass mixing scheme. 

10. Part of the parameter space of sterile neutrino (f/^o? Uro-, Ampg) relevant for the 
LSND/MiniBooNE can be excluded by the the present IceCube 40 data. Namely, the region 
of \Uhq\ ~ I^^toI > 0.15 and Am^ > 0.8 eV^ is excluded at about 3(j level. At the same time in 
certain regions of this parameter space, e.g. |f/ro| ~ 0, |f/^o| = 0.13 — 0.27 (which correspods 
to the flavor mixing) one can obtain even better flt of the data than in the no z/^- mixing 
case. Our analysis has an illustrative character and complete scan of the whole parameter 
space is beyond the scope of this paper. Such an analysis can be done after release of new 
IceCube data and better understanding of systematic errors. Substantial improvement of 
sensitivity to sterile neutrino oscillations will be possible when the two dimensional (energy- 
zenith angle) distribution of events will be available [20]. That is, future studies of the 
zenith angle distributions with different energy thresholds or in different energy intervals 
will allow to perform very sensitive search for sterile neutrinos. 
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Appendix. Constant density case 

To a good approximation the case of constant density can be applied for neutrinos crossing 
the mantle of the Earth only. For constant the Hamiltonian ([s]) can be diagonalized by 
the rotation on the mixing angle in matter am'- 

r, sin^ 2a , , 

sm"'2am = 9 ■ (44) 

cos 2a — -r-V + sm 2a 
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Integration of the evolution equation is then trivial, giving the S*— matrix 



Sn 
















(45) 



e- 

where Him are the eigenvalues of the Hamiltonian in matter: 



_1/Am23 \ Ami, 



cos 2a 



+ sin 2a 



(46) 



Him corresponds to the + sign. In the antineutrino (resonance) channel the eigenvalues, as 
functions of neutrino energy, have the following asymptotics: 
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(4J 



M 2E ^ •^Z^- 

Since for antineutrinos > 0, one has if 2m < 0. In the limit of high energies: Him — H2m 
Returning back to the u basis, 5* = U{am)SmU^ {cim)^ we obtain 

,2 „ „—iH-]mX I „„„2 



yl^/^/ = sin^ amC'^^"''' + cos^ a^e"*^"'"''. 



Then insertion of this amplitude in (14) gives 



P, 



MM 



sin^ ^23 (sin^ a^e"*^^'"^ + cos^ a^e-^^^™^) + cos^ 6236"^'^ 



(49) 



(50) 



and explicitly: 



Pfifi = 1 - sin^ 6123 sin^ 2am sin^(0i - ^2) - sin^ 26'23(sin^ am sin^ 0i + cos^ sin^ 02)- (51) 
Here 

1 



1 / Am^ 

-(iil^X + 032) = 2 + 



X, 



and consequently. 



-(ii2mX + 032) = 2 I ^2m + 1 ^' 



^1 — 02 = ^{Him — H2m)x. 



(52) 



(53) 
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If — )■ (vacuum oscillation limit), — ^ and Him — ^ ~^e^- Therefore 

01 ^^a:, 02 ^^x, 01-0. = (54) 
In this case also am oi and the averaged over fast oscillations probability equals 

— /\ TT)'^ Of] 

p^^ = l- cos^ a sin^ 2^23 sin^ 0.5 sin^ ^23 sin^ 2a - 0.5 sin^ 2^23 sin^ a. (55) 

The first two terms correspond (up to cos^ a) to the standard 2-3 probability and corrections 
are of the order sin^ a. 

In the limit of high energies for antineutrinos we have Him sin^ a and if.m 



K\- So, 



1 / Amnq . 9 . Am|2\ AmpgX . 2 



01 = - ^^sin^a + X ^ ^^^sin'a, (56) 

^ 2 V 2E 2E J AE ' ^ ' 

(t)2 = \{-K\x + (l)-i2)^-\\V^,\x. (57) 
For high energies (above the resonance): am ~ 90°, and consequently, 

C /\ 771^ OC \ 
~^E~ " ■ ^^^^ 

In the case of constant density the p'^ — v'^ probability is described by usual oscillation 
formula: 

Pr'r' = 1 — sin^ 2am sin^ 0m, (59) 
where 0^ is the half-phase of oscillations in matter: 



AmQgX 



A I COS 2a — , „ + sin 2a. (60) 

AE \ \ AmlJ 
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10^ 10^ 

E (GeV) 



Figure 1: The survival probability of the muon antineutrinos (resonance channel) as func- 
tion of the neutrino energy for different values of the zenith angle (cos 6'^) and oscillation 
parameters (Arrigg, sin^a). 
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Figure 2: The same as in Fig. [T]for muon neutrinos. 
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3.0 3.2 3.4 3.6 3.8 4.0 
Logio[£, (GeV)] 

Figure 3: The energy spectrum of integrated over the zenith angles in the intervals 
97° — 180° and 124° — 180° with and without oscillations to sterile neutrinos versus IceCube 
result. We use the z/^— mass mixing scheme with sin^a = 0.08 and Amos = 1 eV^- In the 
top panel, the error bars denoted by the dashed lines include both statistical and systematic 
errors as reported by Icecube (Table II in [13]). In the lower panels, statistical-only error 
bars are denoted by solid vertical lines. For the top panel, the statistical error bars are 
about the same size as the points. 
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Logio(£,/GeV) 

Figure 4: The energy spectrum integrated over the zenith angle with and without oscil- 
lations to sterile neutrinos versus the IceCube results. We use sin^a = 0.08 and Arripg = 1 
eV^. 
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Figure 5: The z/^ energy spectrum integrated over different intervals of zenith angles for 
different values of sin^a and Aml^- 
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Figure 6: The zenith angle dependence of the suppression factor of the muon events inte- 
grated over the energy from Eth = 0.1 TeV. We use the i/^— mass mixing scheme. 



30 



1.00 



0.95 



0.90 



o 
-I— > 

c 

'(/I 



0.85 ; 
^ 0.80 




Am53 = 0.5 eV^ 



sin^ a 



0.02- 
0.04- 
0.08- 



o 



l.OOr 



0.95- 



0.90 




H 0.85 

Oh 

^ 0.80- 



Am; 



03 



0.02 

0.04 

0.08 



in 



1.00 
0.95 
0.90 
0.85 
0.80 



























sin^ ff 






0.02 






0.04 




- Amgj = 2 eV^ 


0.08 











-0.8 -0.6 -0.4 

Cos(0J 



-0.2 



Figure 7: The same as in fig. [6] for Eth = 1 TeV. 
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Figure 8: The zenith angle distribution of muons from z/^ interactions integrated over the 
energy with and without oscillations (solid black histograms) to sterile neutrinos. We have 
renormalized the event distribution according to the best-fit normalization and tilt param- 
eters from the fit (Table 1). Also shown are the IceCube results. 
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Figure 9: Bounds on the active 
IceCube zenith angle distribution of events. Left panel - statistical errors only, right panel: 
statistical plus 5% uncorrelated systematic errors in each bin. 
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Figure 10: The same as in Fig. [Tjfor the Vs ~ i^fi mixing scheme. 
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Figure 12: The same as in Fig. |6|for the Ug — mixing scheme 
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gure 13: The same as in fig. [7] for tlie — z/^ mixing sclieme 
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Figure 14: The zenith angle distribution of muons from z/„ interactions integrated over the 



energy with oscillations to sterile neutrinos in the Us 



mixing scheme. We have renor- 



malized the event distribution according to the best-fit normalization and tilt parameters 
from the St (Table 2). Also shown are the IceCube results. 
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E (GeV) 

Figure 15: The survival probabilities in the antineutrino (resonance) channel as functions of 
neutrino energy for different mixing schemes in the leading order approximation. The top 
and bottom panels show the probabilities for different values of sin^ /3 {s\^ and two different 
zenith angles, while keeping fixed. 
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Figure 16: The suppression factor in the leading order approximation as function of the 
zenith angle for fixed 624 and different values of sin^/3. We used Eth = 0.1 TeV. 
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